Lecture 15

Bias and Least Squares Regression

Review

Least Squares Regression

Regression, generally, is choosing a function of covariates $X_i$ to predict outcomes $Y_i$.
We choose from a set of functions $\mathcal{M}$ that we call a regression model.

\[ Y_i \approx \hat\mu(X_i) \qfor \hat\mu \in \mathcal{M} \]

Least Squares Regression is a special case in which we choose by minimizing the sum of squared residuals.

\[ \hat \mu = \mathop{\mathrm{argmin}}_{m \in \mathcal{M}} \sum_{i=1}^n \qty{ Y_i - m(X_i) }^2 \]

Given a model, we can find the least squares estimator by doing this minimization.
And we can do this for any model we like, so long as we can solve the minimization problem. \[ \begin{aligned} \textcolor{red}{\mathcal{M}} &= \qty{ \text{all functions} \ m(x) } \\ \textcolor{blue}{\mathcal{M}} &= \qty{ \text{all lines} \ m(x)=ax + b } \\ \textcolor{magenta}{\mathcal{M}} &= \qty{ \text{all increasing functions} \ m(x) } \\ \end{aligned} \]

Linear Models

In this class, we’re going to focus on linear models.
- These are sets of functions that are ‘closed’ under addition, subtraction, and scaling by constants.
- This means when we add, subtract, and scale functions in the model, we get another function in the model. \[ a(x), b(x) \in \mathcal{M}\implies a(x) + b(x), a(x) - b(x), c a(x) \in \mathcal{M}\quad \text{ for any constant $c$ } \]
Q. Which of these models are linear?

\[ \begin{aligned} \textcolor{red}{\mathcal{M}} &= \qty{ \text{all functions} \ m(x) } \\ \textcolor{blue}{\mathcal{M}} &= \qty{ \text{all lines} \ m(x)=ax + b } \\ \textcolor{magenta}{\mathcal{M}} &= \qty{ \text{all increasing functions} \ m(x) } \\ \end{aligned} \]

The Additive Bivariate Model

Here we let our predictions be arbitrary functions of education ($x$) as before.
But we’re requiring that race ($w$) only shift that function up and down—it can’t change the function’s shape.
The result, in visual terms, are predictions for different values of $w$ are ‘parallel curves’.

\[\small{ \begin{aligned} \textcolor{blue}{\mathcal{M}} &= \qty{ m(w,x) = m_0(w) + m_1(x) \ \ \text{for univariate functions} \ \ m_0, m_1 } &&\text{additive bivariate model} \\ \end{aligned} } \]

The Parallel Lines Model

Here we impose the additional restriction that the function of education ($x$) is a line.
This means that the predictions for different values of $w$ are parallel lines.

\[\small{ \begin{aligned} \textcolor{red}{\mathcal{M}} &= \qty{ m(w,x) = a(w) + b x } &&\text{parallel lines} \\ \end{aligned} } \]

The Not-Necessarily-Parallel Lines Model

Here we stick with the lines thing, but drop the additivity restriction.
For each value of $w$, we can get a totally different line.

\[\small{ \begin{aligned} \textcolor{magenta}{\mathcal{M}} &= \qty{ m(w,x) = a(w) + b(w) x } &&\text{lines} \\ \end{aligned} } \]

Where This Leaves Us

Each of these approaches allows us to make predictions everywhere—even in columns we haven’t sampled.
Fundamentally, it’s because they’re using information from one column to make predictions in another.
This solves the empty-groups problem, but it introduces a new one. What is it?

Bias

What It Is

Bias is the difference between an estimator’s expected value and what we mean to estimate.
Intuitively, it’s the systematic part of our estimator’s error.
Visually, it’s the difference between …
- The location of our estimator’s sampling distribution. The black line.
- The location of what we’re trying to estimate. The green line.

Q. What is bias here?

It’s the difference in the location of the black and green lines: $\text{bias} = 1 - \textcolor{green}{0} = 1.$

Bias and Consistency

In Practice Midterm 1, we looked at the sampling distribution of three estimators at three sample sizes.
- Estimator a was unbiased.
- Estimator b was biased, but its bias decreased with sample size.
- Estimator c was biased and its bias didn’t change with sample size.
Constant bias means our estimator isn’t going to much closer to the truth as we get more data.
We call estimators like this—estimators that don’t get arbitrarily close to the truth in large samples—inconsistent.

A Biased Least Squares Estimator

\[ \begin{aligned} \textcolor{red}{\mathcal{M}} &= \qty{ \text{all functions} \ m(x) } \\ \textcolor{blue}{\mathcal{M}} &= \qty{ \text{all lines} \ m(x)=ax + b } \\ \end{aligned} \]

We can see that, when we use the all lines model, our least squares estimate $\hat\mu(8)$ is biased.
Why? Because the best line, trying to get the slope right in the middle where most of the data is, gets it wrong at the edges.
Typically, we can’t know an estimator is biased, but we can suspect it is.
- e.g., because it’s nowhere near the subsample means.
That’s because bias is a property of the estimator’s sampling distribution. And we can’t observe that directly.
But in this case something gives it away: $\hat\mu(8)$ is a negative estimate of a positive quantity.

Bias and Fake Data

To really see bias, we have to work with fake data.
We’ll work with a fake version of the earnings data we’ve been using.
- Earlier in the semester, we’ve used a fake version of our income data for illustration.
- Now we’re actually going to sample from it.

fake.sample = fake.pop[sample(1:nrow(fake.pop), size=n, replace=TRUE), ]

And we can actually calculate our estimator’s error.

error = mean(fake.pop$income[fake.pop$education==8]) - muhat(8)

And therefore its bias, the mean of our estimator’s error over many samples.

Bias and Fake Data

We can see, looking at our estimator’s sampling distribution, that it has downward (i.e. negative) bias.

muhat8.sampling.distribution = replicate(1000, {
  fake.sample = fake.pop[sample(1:nrow(fake.pop), size=n, replace=TRUE), ]
  model = lm(income~education, data=fake.sample)
  predict(model, newdata=data.frame(education=8))
})

Q. Does this go away with more data?

It Does Not

muhat8.sampling.distribution = function(sample.size) { 
  replicate(1000, {
    fake.sample = fake.pop[sample(1:nrow(fake.pop), size=sample.size, replace=TRUE), ]
    model = lm(income~education, data=fake.sample)
    predict(model, newdata=data.frame(education=8))
  })
}

dist.quartern   = muhat8.sampling.distribution(round(n/4))
dist.n          = muhat8.sampling.distribution(n)
dist.fourn      = muhat8.sampling.distribution(4*n)

What This Is About

Bias doesn’t go away with more data. What we’re seeing here is …
- In blue, our model’s fit to the population.
- In cyan, its fit to a sample of size $n/4$.
- In magenta, its fit to a sample of size $n$.
- In orange, its fit to a sample of size $4n$.
As sample size increases, we get closer to the blue line.
But the blue line isn’t close to the subpopulation (i.e. column) means $\textcolor{red}{\hat\mu(x)}$.

Least Squares Bias

This isn’t great, but at least we know what’s going on.
In fact, we can prove that the bias of the least squares estimator $\hat\mu(8)$ …
- is the error we get when we fit the model to the whole population
- no matter what sample size we’re using
As sample size increases …
- our estimator’s sampling distribution gets narrower and narrower
- but—if we have bias—it get narrower around the wrong value.

It’s Not Just About Lines

Additive bivariate model fit to a sample from the fake population.

Additive bivariate model fit to the fake population itself.

When we use the additive bivariate model to predict income from education and race …
- some subpopulation means are still far from the model’s predictions
- i.e. some predictions $\textcolor[RGB]{0,0,255}{\hat\mu(x)}$ are still biased estimators of $\textcolor[RGB]{248,118,109}{\mu(x)}$

But Lines Aren’t Great

Additive bivariate model fit to a sample from the fake population.

Additive bivariate model fit to the fake population itself.

That said, when we use the additive lines model to predict income from education and race, we tend to do worse.
- The misspecification, or ‘wrongness’, of a model is a matter of degree.
- And this one is more wrong.

Bias and Summaries

Two Versions of Our MHT Experiment

Researchers hypothesize that MHT increases bone density. We want to know is how well it works, on average, for the people who take it.
Above, we’ve got a population of 10K people eligible for some form of MHT randomized two ways.
- Their age at the time the study begins is on the $x$-axis.
- Bone density, measured 2 years later, is on the $y$-axis.
- Color indicates whether they did or didn’t do MHT.
- On the left, randomization probabilities depend on age. On the right, they don’t.
How does the way we randomize affect the bias of our estimator when we use a bad model?

Version 1: Age-Dependent Randomization

\[ \begin{aligned} \textcolor[RGB]{0,191,196}{\frac{1}{m_1}\sum_{j:w_j=1}} \qty{ \textcolor[RGB]{0,191,196}{\hat\mu(1,x_j)} - \textcolor[RGB]{248,118,109}{\hat\mu(0,x_j)} } &= 0.667 - 0.501 = 0.166 && \text{all functions} \\ &= 0.683 - 0.56 = 0.122 && \text{additive} \\ &= 0.68 - 0.564 = 0.116 && \text{lines} \\ \end{aligned} \]

Version 2: Age-Independent Randomization

\[ \begin{aligned} \textcolor[RGB]{0,191,196}{\frac{1}{m_1}\sum_{j:w_j=1}} \qty{ \textcolor[RGB]{0,191,196}{\hat\mu(1,x_j)} - \textcolor[RGB]{248,118,109}{\hat\mu(0,x_j)} } &= 0.665 - 0.504 = 0.162 && \text{all functions} \\ &= 0.665 - 0.503 = 0.162 && \text{additive} \\ &= 0.665 - 0.503 = 0.162 && \text{lines} \\ \end{aligned} \]